Distributed collaborative knowledge management for optical network

Network automation has been long time envisioned. In fact, the Telecommunications Management Network (TMN), defined by the International Telecommunication Union (ITU), is a hierarchy of management layers (network element, network, service, and business management), where high-level operational goals propagate from upper to lower layers. The network management architecture has evolved with the development of the Software Defined Networking (SDN) concept that brings programmability to simplify configuration (it breaks down high-level service abstraction into lower-level device abstractions), orchestrates operation, and automatically reacts to changes or events. Besides, the development and deployment of solutions based on Artificial Intelligence (AI) and Machine Learning (ML) for making decisions (control loop) based on the collected monitoring data enables network automation, which targets at reducing operational costs. AI/ML approaches usually require large datasets for training purposes, which are difficult to obtain. The lack of data can be compensated with a collective self-learning approach. In this thesis, we go beyond the aforementioned traditional control loop to achieve an efficient knowledge management (KM) process that enhances network intelligence while bringing down complexity. In this PhD thesis, we propose a general architecture to support KM process based on four main pillars, which enable creating, sharing, assimilating and using knowledge. Next, two alternative strategies based on model inaccuracies and combining model are proposed. To highlight the capacity of KM to adapt to different applications, two use cases are considered to implement KM in a purely centralized and distributed optical network architecture. Along with them, various policies are considered for evaluating KM in data- and model- based strategies. The results target to minimize the amount of data that need to be shared and reduce the convergence error. We apply KM to multilayer networks and propose the PILOT methodology for modeling connectivity services in a sandbox domain. PILOT uses active probes deployed in Central Offices (COs) to obtain real measurements that are used to tune a simulation scenario reproducing the real deployment with high accuracy. A simulator is eventually used to generate large amounts of realistic synthetic data for ML training and validation. We apply KM process also to a more complex network system that consists of several domains, where intra-domain controllers assist a broker plane in estimating accurate inter-domain delay. In addition, the broker identifies and corrects intra-domain model inaccuracies, as well as it computes an accurate compound model. Such models can be used for quality of service (QoS) and accurate end-to-end delay estimations. Finally, we investigate the application on KM in the context of Intent-based Networking (IBN). Knowledge in terms of traffic model and/or traffic perturbation is transferred among agents in a hierarchical architecture. This architecture can support autonomous network operation, like capacity management.La automatización de la red se ha concebido desde hace mucho tiempo. De hecho, la red de gestión de telecomunicaciones (TMN), definida por la Unión Internacional de Telecomunicaciones (ITU), es una jerarquía de capas de gestión (elemento de red, red, servicio y gestión de negocio), donde los objetivos operativos de alto nivel se propagan desde las capas superiores a las inferiores. La arquitectura de gestión de red ha evolucionado con el desarrollo del concepto de redes definidas por software (SDN) que brinda capacidad de programación para simplificar la configuración (descompone la abstracción de servicios de alto nivel en abstracciones de dispositivos de nivel inferior), organiza la operación y reacciona automáticamente a los cambios o eventos. Además, el desarrollo y despliegue de soluciones basadas en inteligencia artificial (IA) y aprendizaje automático (ML) para la toma de decisiones (bucle de control) en base a los datos de monitorización recopilados permite la automatización de la red, que tiene como objetivo reducir costes operativos. AI/ML generalmente requieren un gran conjunto de datos para entrenamiento, los cuales son difíciles de obtener. La falta de datos se puede compensar con un enfoque de autoaprendizaje colectivo. En esta tesis, vamos más allá del bucle de control tradicional antes mencionado para lograr un proceso eficiente de gestión del conocimiento (KM) que mejora la inteligencia de la red al tiempo que reduce la complejidad. En esta tesis doctoral, proponemos una arquitectura general para apoyar el proceso de KM basada en cuatro pilares principales que permiten crear, compartir, asimilar y utilizar el conocimiento. A continuación, se proponen dos estrategias alternativas basadas en inexactitudes del modelo y modelo de combinación. Para resaltar la capacidad de KM para adaptarse a diferentes aplicaciones, se consideran dos casos de uso para implementar KM en una arquitectura de red óptica puramente centralizada y distribuida. Junto a ellos, se consideran diversas políticas para evaluar KM en estrategias basadas en datos y modelos. Los resultados apuntan a minimizar la cantidad de datos que deben compartirse y reducir el error de convergencia. Aplicamos KM a redes multicapa y proponemos la metodología PILOT para modelar servicios de conectividad en un entorno aislado. PILOT utiliza sondas activas desplegadas en centrales de telecomunicación (CO) para obtener medidas reales que se utilizan para ajustar un escenario de simulación que reproducen un despliegue real con alta precisión. Un simulador se utiliza finalmente para generar grandes cantidades de datos sintéticos realistas para el entrenamiento y la validación de ML. Aplicamos el proceso de KM también a un sistema de red más complejo que consta de varios dominios, donde los controladores intra-dominio ayudan a un plano de bróker a estimar el retardo entre dominios de forma precisa. Además, el bróker identifica y corrige las inexactitudes de los modelos intra-dominio, así como también calcula un modelo compuesto preciso. Estos modelos se pueden utilizar para estimar la calidad de servicio (QoS) y el retardo extremo a extremo de forma precisa. Finalmente, investigamos la aplicación en KM en el contexto de red basada en intención (IBN). El conocimiento en términos de modelo de tráfico y/o perturbación del tráfico se transfiere entre agentes en una arquitectura jerárquica. Esta arquitectura puede soportar el funcionamiento autónomo de la red, como la gestión de la capacidad.Postprint (published version

Knowledge management in optical networks

©2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Autonomous network operation realized by means of control loops, where prediction from machine learning (ML) models is used as input to proactively reconfigure individual optical devices or the whole optical network, has been recently proposed to minimize human intervention. A general issue in this approach is the limited accuracy of ML models due to the lack of real data for training the models. Although the training dataset can be complemented with data from lab experiments and simulation, it is probable that once in operation, events not considered during the training phase appear thus leading into model inaccuracies. A feasible solution is to implement self-learning approaches, where model inaccuracies are used to re-train the models in the field and to spread such data for training models being used for devices of the same type in other nodes in the network. In this paper, we develop the concept of collective self-learning aiming at improving models error convergence time, as well as at minimizing the amount of data being shared and stored. To this end, we propose a knowledge management (KM) process and an architecture to support it.The research leading to these results has received funding from the European Commission through the METROHAUL project (G.A. nº 761727), from the Spanish MINECO TWINS project (TEC2017-90097-R), and from the Catalan Institution for Research and Advanced Studies (ICREA).Peer ReviewedPostprint (author's final draft

Knowledge management in optical networks: architecture, methods, and use cases [Invited]

© [2019 Optical Society of America]. Users may use, reuse, and build upon the article, or use the article for text or data mining, so long as such uses are for non-commercial purposes and appropriate attribution is maintained. All other rights are reserved.Autonomous network operation realized by means of control loops, where prediction from machine learning (ML) models is used as input to proactively reconfigure individual optical devices or the whole optical network, has been recently proposed to minimize human intervention. A general issue in this approach is the limited accuracy of ML models due to the lack of real data for training the models. Although the training dataset can be complemented with data from lab experiments and simulation, it is probable that once in operation, events not considered during the training phase appear and thus lead to model inaccuracies. A feasible solution is to implement self-learning approaches, where model inaccuracies are used to re-train the models in the field and to spread such data for training models being used for devices of the same type in other nodes in the network. In this paper, we develop the concept of collective self-learning aiming at improving the model’s error convergence time as well as at minimizing the amount of data being shared and stored. To this end, we propose a knowledge management (KM) process and an architecture to support it. Besides knowledge usage, the KM process entails knowledge discovery, knowledge sharing, and knowledge assimilation. Specifically, knowledge sharing and assimilation are based on distributing and combining ML models, so specific methods are proposed for combining models. Two use cases are used to evaluate the proposed KM architecture and methods. Exhaustive simulation results show that model-based KM provides the best error convergence time with reduced data being shared.Peer ReviewedPostprint (author's final draft

Distributed and centralized options for self-learning

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.In general, the availability of enough real data from real fog computing scenarios to produce accurate Machine Learning (ML) models is rarely ensured since new equipment, techniques, etc., are continuously being deployed in the field. Although an option is to generate data from simulation and lab experiments, such data could not cover the whole features space, which would translate into ML models inaccuracies. In this paper, we propose a self-learning approach to facilitate ML deployment in real scenarios. A dataset for ML training can be initially populated based on the results from simulation and lab experiments and once ML models are generated, ML re-training can be performed after inaccuracies are detected to improve their precision. Illustrative numerical results show the benefits from the proposed self-learning approach for two general use cases of regression and classification.This work was partially supported by the EC through the METRO-HAUL project (G.A. nº 761727), from the AEI/FEDER TWINS project (TEC2017-90097-R), and from the Catalan ICREA Institution.Peer ReviewedPostprint (author's final draft

Intent-based networking and its application to optical networks [invited tutorial]

The intent-based networking (IBN) paradigm targets defining high-level abstractions so network operators can define what their desired outcomes are without specifying how they would be achieved. The latter can be achieved by leveraging network programmability, monitoring, and data analytics, as well as the key assurance component. In this tutorial, we introduce the IBN paradigm and its application to optical networking, highlighting the benefits that machine learning (ML) algorithms can provide to IBN. Because the deployment of ML applications requires a specific orchestrator to create ML functions that are connected as ML pipelines, we show an implementation of such an orchestrator. Some challenges and solutions are presented for the generation of accurate synthetic data, proactive self-configuration, and cooperative intent operation. Illustrative examples of intent-based operation and numerical results are presented, and the obtained performance is discussed.The research leading to these results has received funding from the MICINN IBON (PID2020-114135RB-I00) project and from the ICREA Institution.Peer ReviewedPostprint (author's final draft

Combining long-short term memory and reinforcement learning for improved autonomous network operation

A combined LSTM and RL approach is proposed for dynamic connection capacity allocation. The LSTM predictor anticipates periodical long-term sharp traffic changes and extends short-term RL knowledge. Numerical results show remarkable performance.The research leading to these results has received funding from the Spanish MINECO TWINS project (TEC2017-90097-R) and by the ICREA institution.Peer ReviewedPostprint (published version

Supporting beyond 5G applications by coordinating AI-based intent operation: an example for multilayer metro networks

Intent-based Networking (IBN) promises facilitating autonomous decision-making for service assurance. In this paper, we extend IBN by coordinating AI-based intents targeting at supporting beyond 5G services, like immersive and Industry 4.0 applications, on metro infrastructures.The research leading to these results has received funding from the Spanish MINECO TWINS project (TEC2017-90097-R) and from ICREA.Peer ReviewedPostprint (author's final draft

Knowledge management in optical networks: architecture, methods, and use cases [Invited]

© [2019 Optical Society of America]. Users may use, reuse, and build upon the article, or use the article for text or data mining, so long as such uses are for non-commercial purposes and appropriate attribution is maintained. All other rights are reserved.Autonomous network operation realized by means of control loops, where prediction from machine learning (ML) models is used as input to proactively reconfigure individual optical devices or the whole optical network, has been recently proposed to minimize human intervention. A general issue in this approach is the limited accuracy of ML models due to the lack of real data for training the models. Although the training dataset can be complemented with data from lab experiments and simulation, it is probable that once in operation, events not considered during the training phase appear and thus lead to model inaccuracies. A feasible solution is to implement self-learning approaches, where model inaccuracies are used to re-train the models in the field and to spread such data for training models being used for devices of the same type in other nodes in the network. In this paper, we develop the concept of collective self-learning aiming at improving the model’s error convergence time as well as at minimizing the amount of data being shared and stored. To this end, we propose a knowledge management (KM) process and an architecture to support it. Besides knowledge usage, the KM process entails knowledge discovery, knowledge sharing, and knowledge assimilation. Specifically, knowledge sharing and assimilation are based on distributing and combining ML models, so specific methods are proposed for combining models. Two use cases are used to evaluate the proposed KM architecture and methods. Exhaustive simulation results show that model-based KM provides the best error convergence time with reduced data being shared.Peer Reviewe

Modeling and Assessing Connectivity Services Performance in a Sandbox Domain

© 2020 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.The automation of Network Services (NS) consisting of virtual functions connected through a multilayer packet-over-optical network requires predictable Quality of Service (QoS) performance, measured in terms of throughput and latency, to allow making proactive decisions. QoS is typically guaranteed by overprovisioning capacity dedicated to the NS, which increases costs for customers and network operators, especially when the traffic generated by the users and/or the virtual functions highly varies over the time. This article presents the PILOT methodology for modeling the performance of connectivity services during commissioning testing in terms of throughput and latency. Benefits are double: first, an accurate per-connection model allows operators to better operate their networks and reduce the need for overprovisioning; and second, customers can tune their applications to the performance characteristics of the connectivity. PILOT runs in a sandbox domain and constructs a scenario where an efficient traffic flow simulation environment, based on the CURSA-SQ model, is used to generate large amounts of data for Machine Learning (ML) model training and validation. The simulation scenario is tuned using real measurements of the connection (including throughput and latency) obtained from a set of active probes in the operator network. PILOT has been experimentally validated on a distributed testbed connecting UPC and Telefónica premisesThis work was partially supported by the EC through the METRO-HAUL (G.A. nº 761727) project, from the Spanish MINECO/FEDER TWINS (TEC2017-90097-R) and TRÁFICA (TEC2015-69417-C2-1-R) projects and from the Catalan Institution for Research and Advanced Studies

Cooperative learning for disaggregated delay modeling in multidomain networks

© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksAccurate delay estimation is one of the enablers of future network connectivity services, as it facilitates the application layer to anticipate network performance. If such connectivity services require isolation (slicing), such delay estimation should not be limited to a maximum value defined in the Service Level Agreement, but to a finer-grained description of the expected delay in the form of, e.g., a continuous function of the load. Obtaining accurate end-to-end (e2e) delay modeling is even more challenging in a multi-operator (Multi-AS) scenario, where the provisioning of e2e connectivity services is provided across heterogeneous multi-operator (Multi-AS or just domains) networks. In this work, we propose a collaborative environment, where each domain Software Defined Networking (SDN) controller models intra-domain delay components of inter-domain paths and share those models with a broker system providing the e2e connectivity services. The broker, in turn, models the delay of inter-domain links based on e2e monitoring and the received intra-domain models. Exhaustive simulation results show that composing e2e models as the summation of intra-domain network and inter-domain link delay models provides many benefits and increasing performance over the models obtained from e2e measurements.The research leading to these results has received funding from the AEI/FEDER TWINS project (TEC2017-90097-R), from the Catalan Institution for Research and Advanced Studies (ICREA), and from US-EU research collaboration initiative funded by the NSF award #ICE-T:RC 1836921.Peer ReviewedPostprint (author's final draft